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Abstract 

We introduce a spherical Hopfield-type neural network involving neurons 
and patterns that are continuous variables. We study both the thermodynam- 
ics and dynamics of this model. In order to have a retrieval phase a quartic 
term is added to the Hamiltonian. The thermodynamics of the model is ex- 
actly solvable and the results are replica symmetric. A Langevin dynamics 
leads to a closed set of equations for the order parameters and effective corre- 
lation and response function typical for neural networks. The stationary limit 
corresponds to the thermodynamic results. Numerical calculations illustrate 
our findings. 



1 Introduction 

In general, the introduction of continuous versions of discrete spin models in statis- 
tical mechanics has always been welcomed. Since the work of Kac and Berlin pQ 
the spherical model has lead to a better understanding of a lot of basic phenomena 
due to the fact that they simplify the mathematical calculations considerably. In 
spin glass research there is no doubt that an important role has been played by the 
spherical p-spin spin glass model j2] (Hj- Since then it has been quite remarkable 
to see how a mean-field model with spherical spins resembles the results of realistic 
structural glasses (see, e.g., the many references in [I]). 

In neural network theory the Hopfield model is by now a well known classic found 
in many textbooks (e.g., [5], 0) to study storage and retrieval in associative memory 
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problems. Strangely enough, a spherical version of it has not yet been studied in 
the literature. One of the reasons might be that by relaxing the neurons from Ising- 
type to spherical-type variables the retrieval phase is absent, i.e. one of the most 
important properties of this model is lost. 

In this work we show that this problem has an easy and elegant solution. By 
introducing a quartic potential term in the Hopfield Hamiltonian such that the neu- 
rons stay spherical the retrieval phase is recovered. In this way we obtain a simplified 
version of the Hopfield model with an exactly solvable thermodynamics leading to 
marginally replica symmetric results. A phase diagram is obtained with an ex- 
plicit solution for the spin-glass retrieval 'transition' line, showing no re-entrance. 
Moreover, the region of global stability for the retrieval solutions is larger than the 
corresponding region in the standard Hopfield model. 

Next, we study the relaxational Langevin dynamics of this model leading to a 
closed set of equations for the order parameters and effective correlation and response 
functions, typical for neural networks. These equations are similar to those of other 
models with continuous spins (see, e.g, [7j and references therein). We discuss the 
evolution of the overlap order parameter in the retrieval phase. The stationary limit 
of the dynamics is found to correspond to the thermodynamic results. The numerical 
solution of the dynamics illustrates this behaviour. 

The rest of the paper is organized as follows. In section 2 we introduce the model. 
Section 3 contains the thermodynamical properties with the temperature-capacity 
phase diagram. In section 4 we study the dynamics showing the evolution of the 
retrieval overlap. Finally, section 5 summarizes our conclusions. 

2 A spherical Hopfield model 

The Hopfield model jH] is defined through the following mean-field Ising-type Hamil- 
tonian 



where the couplings are related with the information one wants to store in the 
network through the Hebbian rule 



with p = aN, where a is the loading capacity of the network. In this work, the p 
patterns are choosen to be a collection of continuous indepnedent identical random 
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variables (i.i.d.r.v.) with respect to i and \i and drawn from a gaussian distribution 
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and the neurons (spins) are also taken to be continuous and to satisfy the spherical 
constraint 
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As we will show explicitly below this setup of the spherical Hopfield model does not 
allow for a retrieval phase. Therefore, we add the following term to the Hamiltonian 
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which is a quartic term in the order parameter m characterizing the retrieval phase 
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The introduction of this quartic term turns out to contribute macroscopically to 
the condensed part of the free energy but it only leads to sub-extensive contribu- 
tions to the noise produced by the non-condensed patterns and to a sub-extensive 
contribution coming from the diagonal terms, as we now discuss in the following 
section. 



3 Thermodynamic and retrieval properties. 

We apply the standard replica technique in order to study the thermodynamical 
properties of the model defined above. Starting from the Hamiltonian 

= - ^ • / ;/'< n / ~~ JijkiOi(?jVkVi (7) 

i^j i,j,k,l 

and assuming, for simplicity, one condensed pattern, the replicated free energy per 
site becomes 
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In this expression 
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are the retrieval overlap, respectively the neuron (spin) overlap order parameters, 
q a p are conjugate variables and 



Qa/3 — u a 5 a p — g aj a(l — 5, 
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Within a K-th order Parisi replica breaking scheme we assume that the spin 
overlap matrix has the following (ultrametric) structure 



q a/3 = q h if I(a/mi) ^ I(/3/mi) and I(a/m i+1 ) = I((3/m i+1 ) 



(12) 



with {qi}i=o t ...,K a set of real numbers and {mj} i= i r ..^ a set of integers such that 
m i+ i/mi is an integer (m = n,rriK+i = 1), and we introduce the inverse of the 
Parisi function 
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x(q) =n + ) j (m i+1 - rrii)Q(q - qi) . 



(13) 
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In the limit K — > oo, q — > and qx — > qu such that the free energy per site reads 
in the limit n — > 0, iV — > oo 
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where / has to be extremized with respect to x(q) and m. Taking, in general, x(q) 
to be a piece- wise continuous function it is straightforward to show following j3 [TUJ 
that the solution for x(q) reads 



x (q) = 0(q - q M ) 



(15) 
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and, hence, the replica solution is exact. The free energy then reads 



Pf 
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with qu = q, the Edwards-Anderson orderparameter and m 1 = m. 

In agreement with this, one can show by investigating the stability against replica 
symmetry breaking fluctuations that the replicon eigenvalue is zero and, hence, the 
replica symmetric result is marginally stable. 

The saddle-point equations then become simple algebraic equations for q and m 
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with x = /5(1 — q) the susceptibility. We immediately remark that when we remove 
the quartic interaction by setting uq = there exist no solution with m ^ and, 
hence, there is no retrieval possible. 

Henceforth, we take Uq = 1. The corresponding T — a phase diagram is shown in 
Fig. 1. Let us compare this result with the phase diagram of the standard Hopfield 




Figure 1: T — a phase diagram for the spherical Hopfield model. Full (dashed) lines indicate 
discontinuous (continuous) transitions: T$g describes the spin glass transition and Tr ()19fl - H2(Jfl 
indicates the border of existense of the retrieval solutions, T c denotes the thermodynamic transition 
below which the retrieval states are global minima of the free energy. 



model calculated in a replica symmetric approximation Again we have three 



5 



phases. For temperatures above the broken line Tsg, there exist paramagnetic solu- 
tions characterized by m = q = 0, while below the broken line, spin glass solutions, 
m = but q 7^ 0, exist. The transition between the paramagnetic and the spin glass 
phase is continuous and the line T S g separating the two phases is easily computed 

T SG (a) = 1 + (18) 

as in the standard Hopfield model. Next, below the thin full line Tr, (locally stable) 
retrieval solutions, m 7^ and q 7^ 0, appear. The transition between the spin- 
glass and the retrieval phase is, however, a discontinous transition. Using then the 
information that the values of both the order parameters q and m jump when passing 
this line Tr, it is possible to find an analytical expression for the latter. Writing the 
equations (fT7|) as two polynomials in q and m, a jump means that complex roots of 
these two polynomials become real at the same time for all values of a, T obtained 
when crossing Tr. After some algebra this leads to 

a R -sG(P) = [s(/3 - l) 2 /5 2 (7/3 - 16) + S gn(A)-^= + sgn(A)^A|] (19) 
with 

A = 8lV3(/3 - l) 9/2 /3 17/2 (| 125/3 - 128|) 3/2 - A 2 

A 2 = (/3 - l) 5 /3 6 {-2097152 + /3(4849664 + /?[-3459072 + 625/3(1088 + 25/3)])} 
S = (J3 - l) 3 /3 4 {-16384 + 5/3[6144 + /3(-2976 + 125/3)]} . 

(20) 

For T = this line T R leads to the critical capacity a(T = 0) = 4/27 = 0.14814815, 
as seen in Figure 1. 

Below the thick full line, T c , the retrieval states become globally stable. At this 
point we remark that, compared with the standard Hopfield model, there is no re- 
entrance behaviour in the transitions since the spherical model is (marginally) replica 
symmetric. Furthermore, the region of global stability for the retrieval solutions in 
the spherical model is much larger. 



4 Macroscopic Dynamics. 

In this section we study the dynamics of the model. Since the neurons are continuous, 
we assume the relaxational dynamics to be given by the set of Langevin equations 

^ = -tit)*® - 5 ^ + vM + em (21) 
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where the first term on the rhs controls the fluctuations of the neurons, r]i(t) is 
gaussian noise with as first two moments 

(rn(t)) = , (r]i(tH(t')) = 2T8 ij 5{t - t') (22) 

and 6i(t) is an external perturbation field. We discuss this dynamics using the gener- 
ating functional approach ^21 G2] ■ I n a straightforward way the following generating 
functional is introduced 

r N r 

Z[if>}= / D[cr, a] exp / dt^ia^t) + A[tr, a] (23) 

with the action A[cr, tr] given by 

A[<r, Z]=J2[ dt {- Td ^) + (^T + " Ht) + S) ) (24) 

i=l 



where the ipi(t) are the generating fields, (Xj(t) are conjugate variables and D[er, tr] is 
the measure in path space. From this expression all physical quantities of interest can 
be computed by taking derivatives with respect to the generating and the external 
perturbation fields. Assuming that the initial neuron configuration is correlated 
with only one pattern, i.e. the condensed one, and averaging over the disorder 
we arrive, after some algebra, at the following effective single site dynamics in the 
thermodynamic limit 

= -fi(t)a(t)+[m(t) + u m 3 (t)]e + 0(t) + a J dt'[(1L-G)- l G}(t,t')a(t') +<f>{t) 

(25) 

where is a colored noise with mean value and variance 

= , = 2T*(t - + "PI - G^qH - Gt]- 1 ^, t') . (26) 

In this expression m(t) is the retrieval overlap 

m(t) = «£>(*) >*» fl (27) 

and the matrices C and G are the dynamical order parameters of the problem, i.e. 
the correlation and response functions 

C(t,t') = {(a(t)a(t'))J e , G(M') = Q^)(((*(t)U e (28) 
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with (( ■ ■ • )/£! the average over the condensed pattern and ( • • • )^ the average over 
the effective noise 0. At this point we remark that in the course of the calculation 
we see explicitly that the quartic term in the Hamiltonian gives no extensive noise 
contribution. 

Taking the perturbation field 6(t) to zero, using the causality properties of the 
response function and the spherical constraint, C(t, t) = 1, it is possible to write down 
a closed set of equations for the macroscopic observables specifying the dynamics 

d \ f* 

+ fi(t))m(t) = [m(t) + (t)] +a dt'R(t,t')m(t') 



(29) 



dt 

(j^ + fi(t) > )G(t,lf) = 6(t-1?) + aJ i dhRit^Gitut') 
(— + n(t))C(t, t') = 2TG(t', t) + a J dttS(t, ti)G(£', U 

+ [m(t) + u m 3 (t)] m(t') +a f dtxR(t, t x )C{t', t x ) 

J — oo 

where we have defined the effective correlation function, S(t,t'), and response func- 
tion, R(t, £'), as 

S , (t,0 = [I-G]- 1 C[H-G t ]- 1 (t,0, i2(t,0 = [(I-G)- 1 G](t,0- ( 3 °) 

In order to obtain the stationary state from the equations (J2l?j) we assume that, 
close to equilibrium the order parameters become time translation invariant meaning 
that the one-time quantities become time independent and the two-time quantities 
satisfy C(t,tf) = C(t - £'), G(t,t') = G(t - t') (and similarly for R and S). Time 
translation invariance holds when the system is ergodic. Then the correlation and 
response functions are related through the fluctuation-dissipation theorem (FDT) 

pd T C(r) = G{-r) - G{r) , i3d T S{r) = R{-r) - R(r) (31) 

with r = t — t', the initial time — oo and d/dr denoted by d T . With these assumptions 
we can write the evolution equation for the correlation function in the following way 



(d T + n-a/3[l -S(t)])C(t) +a/3 / dt'[S(r - t') - S(r)]d tl C(t') 

Jo 

POO 

= [m + u m 3 ]m + a dt'[R(t' + r)C(t') + S(t' + r)G(t')] . (32) 
Jo 
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With the conditions C(0) = 1 and (3d T C(r) | = —1 we then have for r — ► oo 

C(oo) 



[m + Mom 3 ] 



m 



(3[l-C(oo)} 
Finally, using the evolution equation for m and 

lim C(r) — q, lim S(r) - 



-a(3S(oo)[l-C(oo)}. 



(33) 



[i-/?(i-g)F 



(34) 



it is straightforward to arrive at the equilibrium saddle-point equations (jl7j) . 

At this point we remark that following 3J by starting from the evolution equation 
for the correlation function written as 

(dr + n- a/3[l - S(t)}) /3[1 - C{t)\ - af3 2 [ dt' [S(r - 1') - S(r)] d t ,C(t r ) = 1 (35) 

Jo 

and using the fact that the dynamics is purely relaxational, and hence, that dtC(t) < 
0, d t S(t) < 0, we can derive the following condition for ergodicity 



+ QJ/9- n- fiS(r) > 



(3[1 - C{r)\ 

This inequality seems to indicate that our system is ergodic in the sense of |3j- 



(36) 





Figure 2: The overlap order parameter m(t) as a function of time at temperature T — for several 
values of the capacity: a = 0.041, 0112, 0127from top to bottom on the left and a = 0150, 0160, 0180 
from top to bottom on the right. The horizontal dashed lines correspond to the stationary value 
taken from theory. 
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In fig. 2 we show some plots of the dynamics ()29)1 obtained by discretizing the 
equations. Specifically, the overlap order parameter is shown for several values of 
the loading capacity at temperature zero with m(0) = 1. The behaviour for non- 
zero temperature is qualitatively the same. We clearly see that below the critical 
capacity (figure on the left) the system evolves to the stationary solution, while above 
the critical capacity (figure on the right) m(t) drops to zero. 

5 Conclusions 

In this work we have presented the spherical version of the Hopfield model. In order 
to have a retrieval phase a quartic interaction has been introduced which does not 
destroy the spherical character of the model. The thermodynamic phase diagram is 
qualitatively the same as the one for the standard Hopfield model, except that there 
is no re-entrance because the system is shown to be (marginally) replica symmetric. 
Furthermore, the region of global stability for the retrieval solutions is larger. A 
closed set of equations is obtained for the Langevin dynamics and the stationary 
limit is shown to correspond to the thermodynamic results. A numerical calculation 
of the evolution of the retrieval overlap order parameter illustrates these findings. 
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